Workshop on Optimizing Experimental Designs: Theory, Practice, and Applications

The accurate and efficient measurement of observations is at the core of empirical scientific research. To ensure measurement is optimal, and thereby maximize inference, there has been a recent surge of interest among researchers in the design of experiments that lead to rapid accumulation of information about the phenomenon under study with the fewest possible measurements. Statisticians have contributed to this area by introducing methods of optimizing experimental design (OED: e.g., Atkinson & Donev, 1992; Lindley, 1956), which is related to active learning in machine learning (Cohn, Ghahramani & Jordan, 1996) and to computerized adaptive testing in psychometrics (van der Linden & Glass, 2000). The methodology involves adapting the experimental design in real time as the experiment progresses. Specifically, in OED, an experiment is run as a sequence of stages, or miniexperiments, in which the values of design variables (e.g., stimulus properties, task parameters, testing schedule) for the next stage are chosen based on the information (e.g., responses) gathered at earlier stages, so as to be maximally informative about the question of interest (i.e., the goal of the experiment). OED has become increasing popular in recent years, largely due to the advent of fast computing, which has made it possible to solve more complex optimization problems, and as such is starting to reach everyday experimental scientists. A growing number of labs are applying OED across scientific fields, including cognitive psychology (Myung & Pitt, 2009; Cavagnaro, Myung, Pitt & Kujala, 2010), neuroscience (Lewi, Butera & Paninski, 2009), psychophysics (Lesmes, Jeon, Lu, & Dosher, 2006), systems biology (Kreutz & Timmer, 2009), astrophysics (Loredo, 2004), systems engineering (Allen, Yu & Schmitz, 2003), and clinical drug trials (Wathen & Thall, 2008). OED is not only a useful framework to enhance scientific research, but the underlying principles are also useful as a framework to understand how intelligent agents actively sample information to enhance their learning (e.g., Bramley, Lagnado & Speekenbrink, 2014; Nelson, 2005). The purpose of the workshop is to introduce the principles underlying OED, illustrate how to apply OED in practice using widely and freely available software tools (e.g., R) to showcase applications of OED in areas such as cognitive psychology, education and assessment, and machine learning, and provide a platform to share work on OED.